feat(extract): ingest AsciiDoc documentation (#2938) - #3089
feat(extract): ingest AsciiDoc documentation (#2938)#3089abhay-codes07 wants to merge 2 commits into
Conversation
`.adoc` was in no extension set, so a project documented in AsciiDoc had
none of it in the graph — only the PNGs beside the docs were ingested, and
the detector said so in passing.
`.adoc`/`.asciidoc` join every place a document is decided: DOC_EXTENSIONS
(classification and the semantic pass), the splittable-text set (an
oversized manual is sliced, not truncated), the callflow and hook-guard
extension lists, and markdown's linkable set (a `[text](./guide.adoc)`
link now resolves).
A structural pass, extractors/asciidoc.py, mirrors the markdown extractor
so the deterministic layer is on par: the file is a `page` node carrying
the document title and header attributes (as `frontmatter`), section
titles are `heading` nodes nested by level, and `include::`, `xref:`,
`link:<local path>` and the cross-document `<<file.adoc#anchor>>` form
become `references` edges minted from the resolved path, so they merge
into the linked document's own node (stamped `target_file` for the
incremental remap, like markdown). External URLs, in-page anchors,
non-document targets and paths carrying an unresolved attribute
(`{docdir}/x.adoc`) are skipped rather than guessed; delimited blocks and
comment lines are not parsed for headings or links.
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds an AsciiDoc extractor that mirrors the markdown one: extract_asciidoc turns a .adoc/.asciidoc file into a page node with nested heading nodes per section title, and mints references edges from include::, xref:, link:, and <<...>> targets whose ids match the linked document's own node so they merge rather than spawn ghosts. External URLs, in-page anchors, unresolved-attribute paths, and delimited block bodies (listing/literal/comment/passthrough/example/sidebar/quote) are skipped, and extension-less xrefs resolve to a sibling .adoc. Registers the new extensions across DOC_EXTENSIONS, the extract dispatch, hook source exts, and the HTML viewer's document-type detection, and documents the new format support in the README.
Worth a look
- .asciidoc files are not included in hook source extensions —
graphify/cli.py:74· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Delimited block closes on any same-character delimiter length —
graphify/extractors/asciidoc.py:49· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- _block_delim matches any line of same char length >=4 including heading underlines / horizontal rules —
graphify/extractors/asciidoc.py:52· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2520 functions depend on the 620 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 500 callers, 42 callees - new:
_rebuild_code()— 98 callers, 50 callees - new:
detect()— 110 callers, 15 callees - new:
save_manifest()— 40 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
extract_js()— 83 callers, 3 callees - …and 49 more — each is listed as a finding
Verification — 2520 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 2473 function(s) in the blast radius were not formally verified this run
Formal verification
No difference found (not proven): No behavior difference found in normalize\_node (not a proof).
The verifier ran both versions of normalize\_node on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
· 1 grounded finding(s) anchored inline below; 56 more finding(s) on lines outside this diff (see the check run).
| return Path(os.path.normpath(str(candidate))) | ||
|
|
||
|
|
||
| def extract_asciidoc(path: Path) -> dict: |
There was a problem hiding this comment.
extract_asciidoc()
fans out to 8 callees (efferent coupling); 11 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
…doc in the hook list A nested example block (==== ... ====== ... ====== ... ====) must not close the outer one early, and .asciidoc belongs beside .adoc in the hook-guard source extension list.
|
Follow-up commit on the review's advisory points: a delimited block now closes only on a delimiter of the same character and length (so a nested |
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds a first-class AsciiDoc extractor (extract_asciidoc) that treats .adoc/.asciidoc files as document page nodes, mints heading nodes from section titles nested by level, and emits references edges for local include::, xref:, link:, and <<...>> document targets while skipping external URLs, unresolved attribute paths, and delimited block bodies; edge target ids are minted from the resolved absolute path so they merge into each referenced document's own node. Wires the new extension into _DISPATCH and into the doc-extension sets across detect, cli hooks, and callflow_html node typing so AsciiDoc projects are scanned and rendered as documents rather than ignored. Documents the support in the README.
Worth a look
- AsciiDoc link target can crash extraction via NUL byte —
graphify/extractors/asciidoc.py:133· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- AsciiDoc links can probe and disclose files outside the source tree —
graphify/extractors/asciidoc.py:133· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- AsciiDoc heading IDs collide across same-named files —
graphify/extractors/asciidoc.py:183· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2523 functions depend on the 623 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 500 callers, 42 callees - new:
_rebuild_code()— 98 callers, 50 callees - new:
detect()— 110 callers, 15 callees - new:
save_manifest()— 40 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
extract_js()— 83 callers, 3 callees - …and 49 more — each is listed as a finding
Verification — 2523 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 2476 function(s) in the blast radius were not formally verified this run
Formal verification
No difference found (not proven): No behavior difference found in normalize\_node (not a proof).
The verifier ran both versions of normalize\_node on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
· 1 grounded finding(s) anchored inline below; 56 more finding(s) on lines outside this diff (see the check run).
| return Path(os.path.normpath(str(candidate))) | ||
|
|
||
|
|
||
| def extract_asciidoc(path: Path) -> dict: |
There was a problem hiding this comment.
extract_asciidoc()
fans out to 8 callees (efferent coupling); 12 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Closes #2938.
The problem
.adocwas in no extension set, so a project whose documentation is AsciiDoc had none of it in the graph — the detector reported that only the PNGs beside the docs were ingested._PROSE_EXTSalready knew.adocfor word counting, which is how far it got.The change
.adoc/.asciidocjoin every place a document is decided:DOC_EXTENSIONS— classification, and so the semantic pass digests them like markdown;file_slice._SPLITTABLE_TEXT_SUFFIXES— an oversized manual is sliced, not truncated (Oversized .qmd/.html/.yaml/.skill documents are silently truncated at 20k chars before the semantic pass — only 5 of 10 document types are sliceable #2900's contract);_MD_LINKABLE_EXTS— a[text](./guide.adoc)link now resolves.And a structural pass,
extractors/asciidoc.py, so the deterministic layer is on par with markdown's:pagenode carrying the document title (= Title) and the header attributes (:author:…) asfrontmatter, sanitized like markdown's;==,===, …) areheadingnodes nested by level,file → contains → heading,heading → contains → subheading;include::other.adoc[],xref:other.adoc[...]/xref:other#anchor[](an extension-less xref names a sibling.adoc),link:<local path>[...]and the cross-document<<other.adoc#anchor,text>>form becomereferencesedges. The target id is minted from the resolved path — the same recipe as the target's own file node — so it merges into that node rather than spawning a ghost, and an existing target is stampedtarget_filefor the incremental remap, exactly as markdown does (Incremental rebuild drops cross-file references edges between Markdown docs; GRAPHIFY_FORCE does not help in the hook (follow-up to #1018) #2211).Skipped rather than guessed: external URLs, in-page anchors, non-document targets (
diagram.png), and paths carrying an unresolved AsciiDoc attribute (include::{snippets}/gen.adoc[]). Delimited blocks (----,....,////,++++,====,****,____) and//comment lines are not parsed for headings or links, mirroring markdown's fenced-block handling.What it looks like
Tests
tests/test_asciidoc.py— 21 tests: both extensions present in every set that decides a document (classification, dispatch, slicing, markdown linkability) and listed bydetect(); the page node's title and attributes; section nesting; block and comment skipping; duplicate section titles; a title-less fragment; unreadable files; each link form producing one edge per target; the skip rules; thetarget_filestamp; and a corpus-level check that every reference — in both directions between markdown and AsciiDoc — merges into the linked document's real node with nothing dangling and no stamp leaking.With the registrations reverted and only the extractor module kept, the 4 integration tests fail (the extractor's own unit tests rightly keep passing).
test_detect,test_oversized_document_slicing,test_extractors_registryand the markdown/wikilink suites are unchanged (290 passed); the full suite matches thev8baseline.README's file-types table is updated.